1. Field of the Invention
The present invention relates to a mixed-reality presentation system and a control method therefor, and more particularly, to a mixed-reality presentation system and a control method therefore that enable an observer of mixed-reality space to manipulate a virtual object under observation.
2. Description of the Related Art
Recently, mixed-reality presentation systems that give an observer the sensation of being presented with a mixed reality (MR: Mixed Reality) by composite display of physical space and virtual space seamlessly have come to be widely proposed. These mixed-reality presentation systems generate composite images, in which an image (a virtual space image) rendered using computer graphics (CG) is registered and combined with an image of physical space sensed by an image-sensing apparatus such as a video camera. By generating the composite images according to a position of a viewpoint of the observer and presenting the composite images before the eyes of the observer, the observer is presented with a mixed reality space in which virtual space and physical space are combined. In the presentation of the composite images, a display device typified by a head-mounted display (HMD) is used.
In such systems, in order to present the observer with a sense of mixed reality seamlessly, it is very important to acquire continuously the position and orientation of the viewpoint of the observer, and to generate and display immediately composite images that match any change therein. For this reason, a virtual position and orientation of the viewpoint of the observer is set in virtual space based on the position and orientation of the viewpoint of the observer as measured by position and orientation sensors, and based on this setting an image of a virtual object is rendered using CG and registered with an image of physical space to generate the composite image.
In a mixed-reality presentation system, the observer can observe a composite image in which the virtual object appears actually to exist in physical space. However, in general, the observer can neither move a virtual object under observation nor change its orientation, nor can the observer change the virtual object itself.
In Japanese Patent Laid-Open No. H5-73661, a three-dimensional information conversation system is disclosed in which an operator, using a pen-like stylus 11, manipulates a three-dimensional cursor within a screen and selects, moves and the like a three-dimensional object in the screen. By adding an extension figure to the three-dimensional cursor, JP-A-H5-73661 relieves a problem of the conventional art, in that the positional relation between the three-dimensional cursor in the screen and the stylus 11 that the user manipulates is difficult to grasp and the object to be manipulated is difficult to select.
However, when an attempt is made to apply the technology described in JP-A-H5-73661 to a system that presents an observer wearing an HMD with a mixed reality, a controller like the stylus 11 must be introduced separately. Moreover, unlike the operator in JP-A-H5-73661, an observer wearing an HMD cannot directly observe the controller. Consequently, the correspondence between the position and orientation of the controller, on the one hand, and the position and orientation of the three-dimensional cursor within the composite image on the other, is very difficult to grasp. In other words, in the case of a laser pointer-like input device, because the degree of freedom of manipulation is so high, selection of a desired virtual object, as well as movement and repositioning of the virtual object after it has been selected, is difficult. Moreover, to manipulate the controller during mixed reality presentation means to lose the reality of the mixed reality, which is undesirable.